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Chapter  1 


Introduction 

Current  research  [EEB80]  at  the  University  of  Connecticut  has 
produced  an  experimental  distributed  oomputlng  system  (XDCS1)  which  is 
being  used  to  study  the  structure,  organization,  and  performance  of 
software  to  control  distributed  systems.  The  oomputlng  system  con¬ 
sists  of  four  elanents  or  layers  as  shown  in  the  figure  below: 
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The  outermost  layer  is  the  applications  language  -  in  this  case  EPL 
[MAY].  Below  this  is  the  operating  system  kernel  which  implements  the 
primitive  functions  of  EPL  as  well  as  memory  management.  The  comm un¬ 
cations  subsystem  layer  provides  an  interface  between  the  kernel  and 
the  network  hardware.  The  network,  at  the  lowest  level  of  the  system, 
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consisted  of  five  LSI-11  computers  fully  oonnected  by  serial  links. 

With  the  acquisition  of  a  CSMA/CD  network  baaed  on  ETHERNET 
[DEC80]  the  need  to  create  a  new  communications  subsystem  arose.  Be¬ 
cause  the  CSMA/CD  network  is  used  as  a  packet  switched  network  (as  op¬ 
posed  to  the  circuit  switched,  fully  oonnected,  serial  link  network  of 
XDCS1)  problems  to  be  dealt  with  include:  flow  oontrol  of  packets  In 
the  network,  buffering  of  packets,  and  an  appropriate  communications 
protocol.  Depending  on  the  degree  of  message  transport  reliability 
required,  the  amount  and  complexity  of  the  software  could  become  enor¬ 
mous.  Communications  protocols  based  on  Positive  Acknowledgement  Re¬ 
transmission  (PAR)  schemes  were  considered  as  well  as  a  technique 
which  is  based  on  the  concept  of  virtual  clrult  flow  control 
[SUNSH81 ] . 

Thus  the  major  objective  of  this  thesis  is  to  implement  and  docu¬ 
ment  the  design  of  an  operating  system  which  uses  the  CSMA/CD  channel. 
A  desired  byproduct  of  the  design  is  the  creation  of  a  family  of 
XDCS-like  systems.  The  parameters  which  differentiate  family  members 
are  the  Interprocess  oommuni cations  model  implemented  by  the  kernel 
and  the  underlying  network  and  associated  communications  subsystem 
software.  The  success  with  which  the  family  has  been  created  will  be 
Judged  by  the  extent  software  is  shared  among  family  members  and  the 
ability  to  easily  modify  one  member  to  obtain  another.  Finally  the 
performance  of  the  new  system,  based  on  the  CSMA  channel,  must  be 
evaluated  and  oompared  to  the  other  family  members. 


The  fundamental  characteristics  of  the  XDCS  family  are  described 


In  chapter  two.  Muoh  of  this  discussion  is  based  on  the  work  of  J. 
Morse  [JAM81].  Chapter  three  presents  the  design  of  the  operating 
system  which  was  being  used  at  the  beginning  of  this  experiment 
(XDCS1).  The  reasons  for  this  chapter  are  twofold.  First  it  provides 
the  only  unified  documentation  on  this  version  of  the  operating  sys¬ 
tem.  Another  design  is  fully  documented  in  [PONT60].  The  second  rea¬ 
son  is  to  provide  the  reader  with  the  background  needed  to  understand 
subsequent  design  decisions. 

Chapter  k  deals  mere  specifically  with  design  of  a  new  communica¬ 
tions  subsystem.  The  CSMA/CD  network  will  be  described  in  some  de¬ 
tail.  The  problems  and  solutions  relating  to  implementing  a  commun- 
catlons  subsystem  with  this  network  are  also  discussed  here. 

Chapter  five  oontalns  measurements  which  determine  the  relative 
performance  of  the  CSMA/CD  based  system  (XDCS2)  and  the  serial  link 
system  (XDCS1).  The  effects  of  the  new  communcations  -subsystem 
software  will  be  investigated  as  well  as  any  performance  gains  related 
to  the  increased  bandwidth  of  the  CSMA/CD  channel. 

Chapter  six  is  a  summary  of  the  work  accomplished. 


Chapter  2 


Characteristics  of  the  XDCS  Family 

\ 

2.1  Introduction 

Since  the  XDCS  system  was  already  functioning  at  the  time  of  this 
experiment,  a  design  criteria  was  to  use  the  existing  software  as  much 
as  possible  and  redesign  only  when  necessary.  As  such  this  is  not  a 
formal  attempt  to  create  a  new  family,  but  the  result  should  possess 
many  of  the  characteristics  which  some  researchers  call  a  Family. 

Much  of  the  work  in  the  area  of  families  of  software  systems  has 
been  done  by  Parnas  [PARNS72].  Habermann  [HAB76]  has  applied  these 
techniques  to  the  development  of  operating  systems  in  particular.  In 
the  following  sections  the  design  of  an  operating  system,  which  close¬ 
ly  resembles  the  XDCS  fanily  of  oomputer  systems  is  presented 
[MORSEBO].  These  sections  will  also  provide  an  overview  of  the  actual 
system  upon  which  experimentation  was  done. 
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2.2  The  XDCS  Family 

The  goal  of  this  project  [EEB80]  was  to  develop  an  operating  sys¬ 
tem  for  multiple  loosely-coupled  mini-computers.  Tbe  design  of  the 
system  was  to  be  independent  of  the  type  of  CPU  on  which  the  operating 
system  was  rw,  and  on  the  oommuni cations  facility  available  to  link 
the  CPU's.  Such  a  design  should  make  it  possible  to  implement  a  fami¬ 
ly  of  operating  systems.  The  possible  variations  among  family  members 
included: 

1.  Implementation  for  several  high  level  languages. 

2.  Incorporation  of  new  operating  systems  features  such  as  a  new 
interprocess  communications  model  (IPC). 

3.  Implementation  for  several  different  interconnection  struc¬ 
tures. 

Figure  2.0  illustrates  the  XDCS  family.  Each  path  in  the  tree  defines 

i 

a  family  member.  Each  node  represents  the  software  or  hardware  which 


is  oommon  to  the  fanily  members. 

XDCS  Family 
I 

I  I 

Language(i)  .  Language (n) 

I  I 


Kernel(l)  ....  Kernel(n)  Kernel(l)  ...  Kernel(n) 

lit  I 

Net ( 1 )  Net (m)  Net(1)  Net(m)  Net(1)  Net(m)  Net(1)  Net(m) 

Figure  2.0  -  The  XDCS  Family 

The  current  operating  system  supports  the  programming  language 
EPL  [MAY79].  EPL  is  based  on  the  principle  of  communicating  sequen¬ 
tial  processes  and  is  similar  to  CSP  [HOARE78]  and  Distributed 
Processes  [BRIN78].  Processes  are  autonomous  and  they  communicate  and 
synchronize  with  each  other  only  through  message  passing.  EPL  re¬ 
quires  the  operating  system  to  implement  the  following  primitive 
operations: 

—  SEND  a  message  to  a  process 
—  RECEIVE  a  message  from  a  process 
—  CREATE  a  process 
—  RUN  (initiate)  a  process 

Two  basic  abstractions  to  be  implemented  by  the  operating  system  nu¬ 
cleus  are  processes  and  messages. 
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2.3  Requirements  of  the  distributed  system. 

The  requirements  of  the  operating  system  can  be  summarized  as  follows: 

1.  The  operating  system  kernel  should  support  process  and  mes¬ 
sages. 

2.  Processes  can  do  the  the  following  with  repect  to  other 
processes: 

i.  Create  a  process. 

11.  Run  a  process. 

lii.  Send  a  message  to  another  process. 

iv.  Receive  a  message  from  any  processes. 

v.  Receive  a  message  from  a  specific  process. 

3.  A  process  can  terminate  itself. 

A.  After  a  process  is  created,  there  exists  no  "parent-child" 
relationship  between  the  created  process  and  the  creating  pro¬ 
cess. 

5.  Processes  can  be  viewed  as  finite  state  machines.  They  can 
change  state  independently  of  any  other  process  except  while  exe¬ 
cuting  a  SEND  or  RECEIVE. 

6.  The  result  of  a  RECEIVE  depends  solely  on  the  text  of  the 
message  transmitted  by  a  SEND  operation. 

7.  CREATE  and  RUN  have  no  affect  on  the  state  (local  variables) 
of  the  process  executing  them. 
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8.  The  site  at  which  processes  are  located  has  no  effect  on  the 

operation  of  the  distributed  system. 

Some  of  these  abstract  requirements  can  be  refined  further.  The 
requirement  that  processes  make  Independent  progress  and  may  be  re¬ 
garded  as  Independent  state  machines  suggests  that  the  concept  of  a 
process  splits  into  3  sub-concepts:  1)  each  process  has  a  current 
state.  2)  each  process  has  a  set  of  rules  for  making  transitions  from 
state  to  state  (the  kernel  level  protocol),  and  3)  there  exists  an 
operating  system  dispatcher/ scheduler  which  causes  processes  to  make 
state  transitions. 

The  scheduler  ensures  fair  treatment  to  all  processes  ready  to 
receive  service.  More  importantly  it  provides  one  of  the  fundamental 
goals  of  the  operating  system  -  to  keep  the  CPU  busy  advancing  the 
state  of  runable  processes  whenever  possible.  A  process  descriptor 
contains  all  state  information  associated  with  a  particular  process. 
A  state  transition  diagram  for  the  XDCS1  is  given  in  appendix  2. 

To  Incorporate  processes  located  on  remote  CPUs,  it  is  only 
necessary  to  provide  a  communications  facility  (subsystem)  so  that  the 
remote  processes  can  be  accessed.  All  knowledge  of  the  oommunl cat ions 
hardware  and  system  topology  are  isolated  to  the  communcations  subsys¬ 
tem  (CS).  The  CS  will  determine  whether  the  destination  of  a  message 
is  local  or  remote.  Eventually  the  message  will  be  transmitted  to  the 
destination  process  on  the  appropriate  site.  The  goal  with  this  ap¬ 
proach  is  to  allow  for  changes  in  both  the  communications  facilities 
and  system  topologies  with  no  effect  to  the  other  parts  of  the  operat- 
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ing  system  (i.e  the  operating  system  kernel)  end  to  allow  a  new  IPC 
with  the  same  communications  facilities. 

2.4  Assumptions  made  in  the  XDCS  fanily. 

As  stated  earlier  the  distributed  language  should  be  similar  to 
Hoare's  CSP.  The  language  used  presently  is  EPL.  Other  languages 
which  oould  be  incorporated  in  the  future  inelude:  Distributed 
Processes  [BRIN78]  or  ADA  [DOD80], 

The  kernel  implements  the  primitive  operations  of  the  distributed 
programming  language.  The  implementation  of  these  primitive  opera¬ 
tions  is  determined  by  the  interprooess  communications  model  (IPC)  as¬ 
sumed  by  the  kernel.  The  present  version  of  the  kernel  assumes  that 
each  site  knows  only  about  the  state  of  processes  created  looally. 
Another  assumption  is  that  messages  are  delivered  directly  from  the 
source  process  to  the  destination  process  without  buffering.  The 
latter  assumption  implies  that  a  sending  process  cannot  transmit  a 
message  to  the  receiving  process  until  the  receiving  process  is  ready 
to  accept  that  message.  The  kernel  implements  the  Interprooess  com¬ 
munications  protoools  to  obtain  the  state  of  the  destination  process. 
Another  Important  assumption  which  is  made  by  the  kernel  is  that  all 
messages  are  transmitted  without  error  by  the  communications  subsys¬ 
tem.  Other  IPC  models  may  assume  that  information  about  all  processes 
in  the  system  is  replicated  at  all  sites  or  that  messages  are  buf¬ 
fered.  The  IPC  oould  also  assume  that  the  oommunl cations  subsystem  is 
unreliable  and  that  the  kernel  should  be  prepared  to  handle  oorrupted 


messages  or  failures  in  the  system. 

The  oo mu uncations  subsystem  (CS)  is  st  the  lowest  level  of  the 
XDCS  system.  The  CS  provides  s  virtual  machine  to  the  kernel  by  hid* 
1 ng  the  details  of  the  network.  The  model  of  the  CS  assumes  that: 

1.  the  error  rate  of  the  underlying  network  hardware  ia  very  low. 
Therefore  the  CS  will  not  be  required  to  retransmit  messages 
which  were  corrupted  by  the  hardware  or  a  noisy  environment. 

2.  broadcasting  is  not  required. 

3.  a  protocol  must  be  implemented  to  ensure  that  the  receiver  is 
prepared  to  acoept  a  message. 

<1.  a  protoool  must  be  implemented  to  handle  contention  for  the 
network. 

Different  oommuni cations  subsystems  will  vary  only  in  the  underlying 
hardware  and  the  implementation  of  the  above  model. 
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Chapter  3 


Implementation  of  an  XDCS  operating  system 

In  the  previous  chapter  the  requirements  of  the  family  were  de¬ 
fined.  A  design  which  fulfills  these  requirements  is  now  presented. 

The  software  for  this  Implementation  is  devlded  into  three  lev¬ 
els.  Figure  3.0  shows  the  general  structure  of  the  system.  The  ar¬ 
rows  indicate  the  direction  of  information  flow.  Higher  levels  gen¬ 
erally  use  functions  provided  by  lower  levels. 
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Figure  3.0  -  XDCS  System  Structure 

At  the  top  level  are  the  EPL  processes.  Beneath  this  level  is  the 
kernel.  The  kernel  implements  the  basic  EPL  primitives:  SEND,  REC, 
RECF,  CREATE,  TERMINATE,  and  SYS.  The  kernel  possesses  no  knowledge 
of  the  network  except  for  the  maximum  number  of  sites  in  the  system 
and  its  ixiique  site  number.  At  the  lowest  level  is  the  communications 
subsystem  (CS).  When  one  kernel  needs  to  communicate  with  another 
kernel,  It  submits  a  message  to  the  CS.  The  CS  will  determine  the 
destination  site  and  transmit  the  message  to  the  appropriate  process. 

3.1  EPL  -  Experimental  Programming  Language 

The  current  specification  of  EPL  [EEB80]  consists  of  the  follow¬ 
ing  primitive  operations: 

1.  CREATE  -  CREATE  is  used  by  a  "parent"  process  to  enter  a  new  pro¬ 
cess  ("child"  process)  into  the  system.  An  EPL  process  is  sometimes 
referred  to  as  an  actor.  The  oode  for  the  process  is  called  an  act. 


The  child  Is  allocated  memory  spaoe  on  a  specified  processor.  This 
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Mnary  takes  the  form  of  two  data  structures:  a  process  descriptor 
and  a  data  aegaent  (or  runtine  stack).  The  details  of  these  data 
structures  are  discussed  later. 

2.  HUN  -  The  RUN  primitive  is  used  to  pass  initialization  arguments  to 
the  child  process.  The  child  process  is  then  added  to  a  list  of  ac¬ 
tive  processes. 

3.  FINISH  -  FINISH  allows  a  process  to  terminate  itself  at  arbitrary 
points. 

4.  SEND  -  A  SEND  call  is  made  by  a  process  to  send  a  text  message  to  ( 
or  synchronize  with)  another  process.  A  basic  assumption  made  here  is 
that  messages  are  unbuffered  from  the  process'  point  of  view.  This 
implies  that  the  receiver  must  be  prepared  to  accept  the  message.  If 
the  receiver  is  not  ready  the  sending  process  must  wait. 

5.  RECF  -  If  a  process  needs  information  from  a  specific  process  it 
must  execute  the  RECF  command.  If  the  specified  process  is  not 
prepared  to  send  then  the  receiver  must  wait. 

6.  REC  -  This  command  is  the  same  as  RECF  except  any  sender  is  an  ac¬ 
ceptable  source  of  information. 

7.  SYSTEM  -  SYSTEM  is  for  miscellaneous  operations.  Currently  these 
include:  reading  and  writing  to  terminals,  starting  and  stopping  a 
real  time  clock,  resetting  measurement  variables,  accessing  the  site 
I.D.  and  finding  out  the  current  number  of  sites  in  the  system. 

All  of  the  above  operations  must  be  implemented  by  the  kernel.  Hence 


they  are  also  referred  to  as  kernel  calls.  Before  EPL  executes  a 
kernel  oall  Information  is  placed  in  predetermined  locations.  The 
kernel  functions  know  what  Information  to  expect  and  where  to  find  it. 
Upon  completion,  the  kernel  will  deposit  results  in  locations  known  to 
EPL.  Generally,  information  is  passed  in  the  general  purpose  regis¬ 
ters  (assuming  LSI11  processors).  In  this  way  an  Interface  is  provid¬ 
ed  between  EPL  and  the  kernel. 

The  specific  information  which  is  passed  is  dependent  on  the  ker¬ 
nel  call.  The  description  of  the  information  layout  and  diagrams  for 
each  primitive  are  given  in  Appendix  1. 

3.2  The  Kernel 

In  order  to  implement  the  EPL  primitives  the  kernel  performs  two 
major  functions: 

1.  Resource  management 
and  2.  Process  management 

Resource  management  entails  the  allocation  of  memory  to  newly 
created  processes,  the  naming  of  newly  created  processes,  and  the 
maintanence  of  any  queues  or  lists  in  the  system.  It  also  includes 
the  scheduling  of  processes  for  fair  CPU  utilization.  Process  manage¬ 
ment  involves  maintaning  all  processes  in  a  valid  state. 

An  important  design  decision  lies  in  the  distribution  of  system 
information  [C0HEN80].  Information  ooncerning  the  state  of  processes 
has  been  partitioned  so  that  each  kernel  knows  only  about  local 
processes.  A  kernel  must  make  enquiries  to  other  kernels  for  informa- 
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tlon  about  remote  processes. 

Since  the  kernel  should  possess  no  knowledge  of  a  processes  loca¬ 
tion,  all  messages  are  submitted  to  the  CS  regardless  of  whether  they 
were  local  or  non-local.  This  design  decision  helps  reduce  the  com¬ 
plexity  of  the  kernel  by  not  treating  local  kernel  calls  as  a  special 
case. 


3.2.1  Data  Structures 

Figure  3.1  illustrates  all  of  the  major  data  structures  in  the 
system.  As  stated  in  the  previous  chapter,  when  a  process  is  created 
it  is  allocated  memory  in  the  form  of  two  data  structures:  a  process 
descriptor  and  its  data  segment.  The  process  is  also  given  a  name 
which  is  vnique  in  the  system.  The  name  is  oomposed  of  two  bytes  of 

t 

information: 


15  8,7 

4——————— 

!  index  into  MAP  | 


site  index 


|  15  MAP  0 

I 

+- — >  |  address  | - — >  Actor 


+ _ + 

The  low  order  byte  oontalns  the  CPU  index  where  the  process  was  creat¬ 
ed.  The  high  order  byte  oontains  an  index  into  an  array  called  MAP. 
At  this  location  in  MAP  is  placed  the  base  address  of  the  process 
descriptor.  MAP  contains  the  location  of  every  process  descriptor  lo¬ 
cal  to  its  site.  The  kernel  can  now  use  these  names  to  access  infor- 
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mation  about  any  process  in  the  system. 

Appendix  3  oontalns  a  detailed  layout  of  a  process  descriptor. 
The  size  of  the  process  descriptor  can  vary  with  the  number  of  sites 
permitted  in  the  system  (MAX_SITES).  The  first  word  is  used  to  indi¬ 
cate  the  success  of  the  kernel  call.  Another  slot  holds  the  current 
state  of  the  process.  A  "pending  sender"  field  exists  for  each  site 
in  the  system.  These  slots  hold  the  number  of  messages  which  the 
corresponding  site  would  like  to  send  to  this  process.  Two  slots  ex¬ 
ist  for  linking  process  descriptors  into  lists.  The  last  field  con¬ 
tains  the  name  of  the  process  to  which  that  process  descriptor  be¬ 
longs.  The  data  sequent  of  a  process  is  described  in  detail  in  appen¬ 
dix  1. 

The  kernel  must  maintain  several  queues.  The  local  set  of 
processes,  ready  to  rim  on  the  CPU,  is  contained  in  a  FIFO  queue 
called  the  ready  list.  The  variables  READY. HEAD  and  READY. TAIL  point 
to  the  head  and  tail  of  the  list  respectively.  Processes  which  become 
ready  to  rui  are  placed  on  the  tail  of  the  list.  For  the  current  im¬ 
plementation,  when  the  CPU  beoomes  available  the  next  process  to  use 
it  will  be  taken  off  the  head  of  the  list. 

When  a  process  executes  a  SEND  command  the  kernel  must  determine 
whether  the  destination  process  is  prepared  to  receive  the  message. 
If  the  receiver  is  not  ready  the  sending  process  is  placed  on  a  queue 
of  processes  which  need  to  transmit  messages  to  that  site.  A  queue 
exists  for  each  site  in  the  system.  A  list  of  pointers  (SNDR_Q)  con- 
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tains  the  head  and  tall  of  each  queue  of  senders  -  Figure  3.1. 

At  the  receiving  site,  the  arrival  of  an  enquiry  is  marked  by  In¬ 
crementing  two  fields  in  the  receivers  process  descriptor: 
total^_pending  and  pending_sender [source],  Total_pending  is  the  total 
number  of  processes  at  all  sites  which  have  requested  to  send  a  mes¬ 
sage  to  this  process.  Pending_sender [source]  holds  the  number  of 
processes  which  have  attempted  to  send  a  message  from  a  specific  site. 

Both  the  ready  list  and  the  sender  queue  use  the  two  link  fields 
in  the  process  descriptor  to  create  the  respective  lists.  This  is 
made  possible  be  a  design  invariant  which  permits  a  process  to  be  on 
only  one  queue  at  a  time. 

In  order  to  transport  a  message  the  CS  must  be  given  enough  in¬ 
formation  by  the  kernel  to  do  so.  The  information  needed  is  the  name 
of  the  source  process  and  destination  process,  the  type  of  message  to 
be  transmitted,  and  the  address  of  any  parameters  to  be  transmitted. 
The  kernel  will  place  the  information  in  a  queue  (figure  3.i<)  which 
corresponds  to  the  destination  site.  Each  element  in  the  queue 
represents  a  request  to  transmit  a  message  to  a  remote  site. 

transmit  queue  _ 

- —  |  communications! 

requests - ->  III! - >1  subsystem  I  — >ne  twor  k 


Figure  3,l»  -  Transmitter  queue 


Since  the  information  provided  to  the  CS  includes  process  names, 
the  CS  oust  possess  knowledge  of  the  kernel  data  structures.  The 
number  of  bytes  to  be  transmitted  is  a  function  of  the  message  type. 

The  format  for  each  type  of  message  is  given  in  Appendix  4. 

When  a  message  is  received  by  the  communications  subsystem  the 
inverse  procedure  occurs.  Information  needed  by  the  kernel  is  placed 
in  a  variable  -  CMD.  CMD  has  the  same  structure  as  one  slot  in  the 
transmitter  queues: 

!  source  j  destination  |  function  |  parameters  J 

The  kernel  function  which  corresponds  to  that  message  will  now  use  CMD 
to  complete  or  continue  the  kernel  call  protocol. 

3.2.2  Summary  of  Kernel  Call  specifications 
This  section  describes  the  actions  each  kernel  call  has  to  per¬ 
form  in  order  to  complete  the  EPL  primitive.  The  implementation  of 
each  call  is  dependent  on  the  kernel  level  protocol. 

1.  INIT  -  INIT  is  called  by  a  system  start  up  routine.  In  this  ver¬ 
sion  of  the  kernel  INIT  is  responsible  for  initializing  all  of  the 
system  data  structures.  It  also  initializes  any  I/O  devices  in  the 
system.  If  the  site  index  is  zero,  the  first  EPL  process  is  created 
and  started  running. 

2.  CREATE  -  A  create  call  is  made  by  a  parent  process  to  enter  a  new 
process  into  the  system.  If  the  EPL  program  did  not  specify  the  site 
of  creation,  the  kernel  will  choose  the  site.  The  current  algorithm 
for  choosing  the  site  is  : 

site  of  creation  =  (local  site  index  ♦  1)  modulo  (total  number  of  sites) 
A  create  request  is  made  to  the  chosen  site  which  includes  the  address 
of  the  ACT  (code  which  corresponds  to  the  child  process).  At  the  des- 
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tination  site,  the  kernel  will  allocate  memory  for  the  new  process, 
assign  it  a  unique  name,  and  then  return  the  name  to  the  parent  pro¬ 
cess. 

3.  PUN  -  The  RUN  primitive  is  called  by  the  parent  process  to  initial¬ 
ize  the  child  process.  A  message  is  passed  to  the  child  process  which 
includes  the  number  of  initialization  parameters  to  be  sent  followed 
by  the  parameters  themselves.  The  child  process  will  then  be  placed 
on  its  local  ready  list. 

1».  TERMINATE  -  TERMINATE  is  called  by  a  process  upon  completion  of  its 
code.  Its  state  is  changed  to  indicate  its  terminated  status. 

5.  SEND  -  A  SEND  call  is  made  by  a  process  that  needs  to  transmit  a 
message  to  another  process.  The  "sender's"  kernel  will  transmit  an 
enquiry  to  the  "receiver's"  kernel.  The  receivers  kernel  will  first 
mark  the  presence  of  the  enquiry.  The  kernel  will  then  determine  the 
status  of  the  receiver  process.  If  the  process  is  not  prepared  to  re¬ 
ceive,  a  negative  acknowledgement  is  returned  to  the  senders  kernel. 
At  the  senders  site  the  process  which  initiated  the  SEND  will  be 
placed  on  a  waiting  queue  corresponding  to  the  receivers  site.  If  the 
receiver  is  ready,  a  positive  acknowledgement  is  returned  to  the 
senders  kernel.  The  message  is  then  transmitted  to  the  receiver. 

This  implementation  of  the  SEND  oommand  does  not  allow  the  send¬ 
ing  process  to  execute  until  the  receiver  process  has  accepted  the 
message.  This  implies  that  the  SEND  can  be  used  for  synchronization 
as  well  as  message  transmission. 
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6.  RECEIVE  -  The  RECEIVE  command  la  executed  by  a  process  which  needs 
Information  from  another  process  in  order  to  oontinue.  The  receivers 
kernel  will  first  check  for  enquiries  from  sending  processes  (pending 
senders).  If  there  are  pending  senders,  the  kernel  will  send  a  mes¬ 
sage  which  will  inform  the  sending  kernel  that  the  receiving  process 
is  prepared  to  receive  a  text  message.  At  the  senders  site,  the  ker¬ 
nel  will  determine  if  there  exists  a  process  which  wishes  to  transmit 
a  message  to  the  receiving  process.  The  message  will  be  transmitted 
if  there  is  a  sending  process.  Otherwise  a  negative  acknowledgement 
will  be  returned.  In  the  event  of  a  negative  acknowledgement,  the  re¬ 
ceiving  process  will  be  placed  in  a  waiting  state. 

7.  SYSTEM  -  SYSTEM  is  described  in  appendix  1. 


3.2.3  Process  Management 

A  process  may  be  viewed  as  finite  state  machine  -  Figure  3.5. 
The  next  state  transitions  are  a  function  of  the  EPL  primitive  being 
executed  by  the  source  process,  the  state  of  the  destinations  process, 
and  the  current  state  of  the  source  process. 


inputs : 


Output  is  a  function 
of  the  current  state 


EPL  function - >| 

I  Process 

state  of  destination  process  - >| 

I 

present  state  - >1 


I 

1 

1 


> 


next 

state 


Figure  3.5  -  process  as  a  finite  state  machine 
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The  algorithm  which  governs  the  state  transitions  is  defined  by  the 
kernel  level  protocol.  The  kernel  includes  the  of  routines  which  im- 
pleoent  this  protoool. 

3.2.3. 1  Process  States 

A  state  exists  fcr  each  process  which  corresponds  to  a  step  in 
the  execution  of  a  kernel  call.  The  possible  states  and  their  defini¬ 
tions  for  this  kernel  are  as  follows: 

HrflCfigg  state  Definitions 

1.  CREATED:  the  process  has  been  created  (i.e.  it  has  a  name),  but 
no  parameters  have  been  transmitted  by  the  creating  process. 

2.  CREATING:  the  process  is  creating  a  child  process  and  has  not  yet 
received  the  name  of  the  child  process. 

3.  TRANSMITTING:  the  process  is  either  sending  a  message  to  a  re¬ 
ceiving  process  or  it  transmitting  parameters  to  a  child  process. 

4.  READY:  the  process  has  finished  a  kernel  call  and  is  ready  to  ex¬ 
ecute  but  may  not  be  assigned  a  CPU. 

5.  TERMINATED:  the  process  has  terminated. 

6.  ENQ_REC:  the  process  is  ready  to  receive  a  message  from  any 
sender.  An  enquiry  is  sent  to  the  site  of  a  pending  sender,  will  ei¬ 
ther  be  negative  cr  the  expected  message. 

7.  EN(l_RECF:  the  process  is  ready  to  receive  a  message  from  a 
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specific  Bender.  An  enquiry  is  sent  to  the  site  of  that  sender  (only 
if  there  are  pending  senders  for  this  process).  The  acknowledgement 
will  either  be  negative  or  the  expected  message. 

8.  WAIT_REC:  the  process  is  waiting  to  receive  a  message  and  a  suit¬ 
able  sender  process  has  not  yet  enquired  if  the  process  is  prepared  to 
receive. 

9.  WAIT_RECF :  the  process  is  prepared  to  receive  a  message  from  a 

specific  sender  but  the  sender  is  not  yet  prepared  to  transmit. 

10.  ACK_JENQ:  the  process  has  received  an  enquiry  from  a  potential 
sender  and  has  returned  positive  acknowledgement  to  that  sender. 

11.  ENQ_SEND:  the  process  has  sent  a  message  to  enquire  if  the  re¬ 

ceiver  process  is  prepared  to  accept  a  message. 

12.  WAIT_SEND:  the  process  is  prepared  to  send  a  message  but  the  re¬ 
ceiver  is  not  prepared  accept  it. 

13.  CONCURRENT:  a  process  has  sent  a  message  to  enquire  if  the  re¬ 

ceiver  process  is  prepared  to  accept  a  message.  The  receiver  has  con¬ 
currently  sent  a  message  to  the  sender  process  to  enquire  if  it  is 
prepared  to  transmit  it's  message. 

A  state  transition  diagram  can  be  found  in  appendix  2.  The  specific 
kernel  routines  which  cause  these  transitions  are  identified  and  ex¬ 


plained. 
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3.2. 3.2  Kernel  Level  Protoools 

While  executing  an  EPL  primitive  the  kernel  may  require  informa¬ 
tion  about  processes  on  another  or  the  same  site.  The  protocol  de¬ 
fines  the  oorrect  sequence  of  actions  the  oooperating  kernels  must 
take  in  order  to  effect  the  information  transfer.  Each  state  of  a 
process  corresponds  to  a  different  point  in  the  protocol. 

The  kernel  protocol  is  dependent  on  the  EPL  primitive.  The  fol¬ 
lowing  timing  diagrams  illustrate  each  protocol.  The  format  of  each 
message  of  the  protocol  can  be  found  in  appendix  4. 
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3.2.4  Coma uni cations  Subsystem  for  XDCS1 
One  of  the  criteria  of  the  design  of  the  communications  subsystem 
(CS)  is  to  "hide"  the  details  of  the  network  from  the  kernel.  When 
the  kernel  needs  to  transport  a  message  to  another  or  the  same  kernel, 
it  formats  a  request  to  the  CS,  places  the  request  on  a  transmitter 
queue,  and  then  Informs  the  CS  that  a  request  has  been  made.  The  CS 
will  determine  if  the  message  is  local  or  remote.  If  local,  the  mes¬ 
sage  is  transferred  to  the  local  process.  If  remote,  the  message  must 
be  transmitted  through  the  network  using  a  well  defined  protocol. 

The  CS  can  be  viewed  as  having  three  levels  (Figure  3.6).  At  the 

top  (TRANSMIT)  is  software  for  determining  the  destination  (local  site 

or  remote  site).  This  level  will  also  ensure  that  a  message  will  not 

be  transmitted  until  a  previous  transmission  has  completed.  Below 

this  (NETWORK)  are  the  details  of  the  link  level  protocol.  At  the 

lowest  level  are  the  routines  which  interface  with  the  hardware.  The 

interface  routines  (INPUT  and  OUTPUT)  are  responsible  for  transmission 

and  reception  of  all  messages. 

| - j 

I  TRANSMIT  I 

| - J 

I  NETWORK  I 

j  ■■  _  ,  ,  ,  M  . _ j 

I  INPUT  |  OUTPUT  | 

I - ! 

Figure  3.6.  Structure  of  the  Communications  Subsystem 


30 


3.2.M.1  The  Kernel/CS  interface 

As  stated  In  the  previous  section  a  goal  of  the  CS  is  to  hide  de¬ 
tails  of  the  network.  At  the  sane  tine  it  is  desirable  to  hide  de¬ 
tails  of  the  kernel  from  the  CS.  In  XDCS1  the  kernel  places  informa¬ 
tion  needed  by  the  CS  in  a  transmitter  queue.  The  CS  can  extract  the 
destination  site  and  the  location  in  memory  of  any  parameters  to  be 
transmitted.  Implicit  in  this  approach  is  that  the  CS  possesses 
knowledge  about  the  structure  of  process  descriptors  and  data  seg¬ 
ments  . 

3.2.5  Link  Level  Protocol 

The  link  level  protocol  is  highly  dependent  on  the  network 
hardware  [CQHQJ80] .  The  protocol  in  this  case  assumes  a  set  of  com¬ 
puters  fully  interconnected  by  serial  links. 

The  protocol  is  based  on  the  concept  of  connection  and  is  very 
similar  to  the  protocol  used  in  another  version  [FONT80].  Figure  3.7 
is  a  timing  diagran  of  the  protoool. 
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Figure  3.7  -  Link  Level  protocol 


A  request  for  a  connection  is  passed  from  the  a  source  CS  to  a 
destination  CS.  The  CS  initiating  the  communications  then  waits  until 
the  CS  sends  an  acknowledgement.  Once  a  connection  is  made,  proces¬ 
sors  are  dedicated  to  the  information  exchange  and  remain  connected 
until  all  the  information  has  passed. 

Because  the  basic  protocol  requires  the  initiating  CS  to  wait  for 
and  acknowlegement  from  the  cooperating  CS,  it  is  possible  for 
deadlocks  to  occur.  Another  problem  is  that  it  is  possible  for  each 
of  two  I/O  subsystems  to  simultaneously  attempt  to  initiate  a  connec¬ 
tion  with  the  other.  To  prevent  deadlocks  and  arbitrate  contention, 
the  nodes  are  arranged  in  a  hierarchy  according  to  their  ID  numbers. 
Lower  numbers  have  higher  priority.  As  with  any  hierarchical  scheme, 
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there  is  a  danger  of  starving  the  low  priority  processors. 

The  basic  connection  protocol  has  been  extended  to  work  with  the 
hierarchy.  A  request  for  a  connection  is  passed  from  the  CS  initiat¬ 
ing  the  communications  to  the  CS  it  wishes  to  communicate  with.  The 
initiating  CS  then  waits  for  an  acknowledgement  before  proceeding.  If 
it  is  a  positive  acknowledgement  the  message  is  sent.  If  it  is  a 
negative  acknowledgement,  indicating  that  the  oooperating  CS  is  trying 
to  initiate  communications  on  the  same  link,  its  handling  depends  on 
the  relative  positions  of  the  two  sites.  The  CS  with  lower  priority 
relinquishes  the  line,  sends  a  positive  acknowledgement  and  then  waits 
to  receive  the  message.  After  the  message  has  been  transmitted  the  CS 
with  higher  priority  will  send  a  positive  acknowlegement  to  the  lower 
priority  site  and  then  prepare  itself  to  receive  the  message.  The 
lower  priority  CS  can  then  transmit  the  message. 

While  a  CS  is  attempting  to  make  a  oonnection  without  priority, 
in  order  to  prevent  deadlocking,  it  must  listen  to  kernels  above  it  in 
the  hierarchy.  If  one  of  these  wishes  to  make  a  oonnection,  the  low 
priority  CS  sends  a  positive  acknowlegement  and  receives  the  message 
before  returning  to  its  own  communication.  Once  it  has  made  the  con¬ 
nection,  it  no  longer  listens  to  the  other  communications  subsystems. 


Chapter  i| 
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Communications  Subsystem  Design  for  XDCS2 

As  was  stated  in  the  introduction  to  this  thesis:  the  primary  ob¬ 
jective  was  to  design  a  new  communications  subsystem  based  on  a 
CSMA/CD  network.  Before  a  specific  design  is  undertaken  the  general 
structure  of  communications  subsystems  will  be  examined. 

The  model  for  a  communications  subsystem  for  the  XDCS  system  is 
discussed  in  Chapter  two.  Implementations  of  this  model  can  vary  with 
respect  to  the  services  provided  by  the  oommuni cations  subsystem  and 
the  underlying  network. 

Recent  efforts,  by  the  ISO  to  standardize  the  decomposition  of 
abstract  network  functions  [IS079],  list  the  services  which  should  be 
provided  by  CS  software.  A  summary  of  their  results  will  be  the  first 
topic  of  this  chapter. 

It  will  be  shown  that  the  services  needed  for  a  serial  link  net¬ 
work  are  slightly  different  than  for  a  CSMA/CD  network.  After  the 
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services  required  by  the  CSMA/CD  are  determined,  a  specific  implemen¬ 
tation  will  be  described. 

4.1  Abstract  Communications  functions  -  ISO  Model 

Since  a  distributed  oomputing  system  may  be  viewed  as  a  local 
area  network  some  of  the  techniques  used  to  communicate  between  nodes 
in  a  network  may  be  used  here.  In  particular,  the  ISO  has  attempted 
to  use  layering  to  define  a  communications  subsystem  architecture 
[ ZIM80 ] . 

The  proposed  layers  are  shown  in  figure  4.0.  Each  layer 

represents  a  protocol  or  format  which  is  needed  to  perform  the  func¬ 
tion  of  that  level. 

I  Application  ! 

!  Presentation  i 

!  Session  i 

i  Transport  i 

!  Network  | 

!  Link  i 

!  Physical  i 


Figure  4.0  -  Open  System  Architecture 

At  the  lowest  level  are  the  physical/electrical  standards.  For 
example  a  distributed  processing  system  may  require  each  node  to  use 
the  same  kind  of  serial  link  interface.  The  specification  of  all  eon- 
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nector  leads,  voltages,  etc  are  also  provided  at  this  level. 

The  link  level  is  responsible  for  moving  raw  data  over  the  net¬ 
work  reliably.  The  link  level  may  have  some  notion  of  the  data  format 
but  it  will  use  this  knowledge  for  low  level  functions  only  -  e.g. 
error  detection  and  correction.  This  should  be  the  only  software  in 
the  system  that  interacts  directly  with  the  network  hardware. 

The  network  level  is  responsible  for  message  transfer  between 
nodes  in  the  network.  This  level  could  also  be  referred  to  as  the 
switching  level.  It  establishes  connections,  real  or  "virtual", 
between  two  or  more  sites.  Routing  functions  are  also  provided  here. 

The  transport  layer,  in  combination  with  the  lower  levels,  should 
comprise  a  oomplete  communications  subsystem.  Services  provided  here 
could  Include:  flow  control,  end  to  end  error  control,  sequencing  of 
messages,  and  duplicate  message  detection.  Provisions  for  virtual 
circuits  should  also  be  given  here. 

The  remaining  three  levels:  Applications,  Presentation,  and  Ses¬ 
sion,  provide  for  communication  at  the  process  level.  In  the  XDCS 
family  the  EPL  and  kernel  layers  correspond  to  these  levels.  These 
higher  levels  should  possess  no  knowledge  of  the  system  topology.  It 
should  be  noted  that  the  ISO  standard  only  provides  a  guideline  for 
decomposing  network  functions.  Other  networks  have  been  constructed 
using  the  layering  concept  which  do  not  fit  precisely  into  this  model. 
Examples  include  IBM's  SNA  [GRAY79],and  Digital  Equipment  Corporations 
DNA  [WECXER80] .  Another  example  is  the  communications  subystem  for 
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XDCS1. 

4.2  Abstract  functions  required  by  a  serial  link 

The  impl ementation  of  a  communications  subsystem  based  on  serial 
links  (XDCS1),  is  presented  in  chapter  3.  This  section  now  describes 
the  relationship  of  XDCS1  to  the  ISO  model  of  communications  subsystem 
architecture. 

It  was  stated  in  chapter  3  that  the  CS  for  a  serial  link  network 
used  only  one  protocol:  the  "link  level"  protocol.  In  terms  of  the 
ISO  model  this  link  protocol  provides  the  functions  of  two  ISO  levels: 
the  link  level  and  the  network  level.  The  only  link  level  service 
provided  is  raw  data  transmission.  No  error  detection  or  correction 
is  provided.  The  network  level  establishes  an  hierarchy  among  sites 
in  the  system  and  makes  connections  between  the  sites.  No  message 
routing  is  performed. 

There  is  no  logical  equivalent  of  a  transport  layer  in  this  sys¬ 
tem.  This  is  not  a  problem  because  there  is  no  need  for  the  functions 
provided  by  this  level.  Flow  control  is  effectively  accomplished  by 
the  connection  protocol  used  in  the  network  level. 

4.3  The  CSMA/CD  network. 

The  network  used  in  XDCS2  is  an  example  of  a  shared  bus  which 
uses  a  carrier  sense  multiple  access  with  collision  detect  (CSMA/CD) 
communcations  protocol  [TOBAG79J,  [SWARTZ77].  The  hardware  used  by 
the  network  consists  of  a  ooaxial  cable  and  a  set  of  Network  Interface 
(NI)  boards  [DEC80].  Each  site  in  the  system  requires  the  use  of  one 


NI  board  to  be  part  of  the  system  -  Figure  U .1 
ooaxial  cable 
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Figure  H.l.  HI  based  network. 

The  properties  of  the  network  are  very  similar  to  Ethernet-like  sys¬ 
tems  [METCALF76] .  Each  station  in  the  network  can  oommunicate  by 
serially  transmitting  packets  of  information  over  the  cable  to  any 
other  station  which  has  tapped  the  cable.  The  packet  contains  desti¬ 
nation  address(s)  and  is  copied  from  the  cable  by  that  destinatlon(s) . 
The  destination  site  must  specify  the  address  of  a  buffer  into  which 
messages  can  be  placed. 

Control  and  access  to  the  cable  is  completely  distributed  among 
all  stations.  Each  NI  board  provides  the  facilities  to  implement  the 
CSMA/CD  link  level  protocol.  Transmissions  initiated  by  a  site  defer 
to  any  which  may  already  be  in  progress.  Once  started,  if  interfer¬ 
ence  with  other  packets  (collision)  is  detected  a  transmission  is 
aborted.  After  a  collision  has  been  detected  the  transmitting  NI 
will  "jam"  the  cable  thereby  ensuring  that  all  sites  involved  in  the 
transmission  will  detect  the  oolllsion.  The  transmitting  site  can 
then  retransmit  the  message  if  it  so  desires. 
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Some  Important  characteristics  and  specifications  which  may  con- 
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cern  users  of  the  NI  boards  Include: 

1.  After  detecting  that  the  cable  is  idle  the  NI  board  will  wait 
a  prespecified  time  period  before  transmitting  a  message.  The 
current  design  waits  256  microaeoonds. 

2.  The  user  must  provide  the  address  of  a  buffer  into  which  a  re¬ 
ceived  message  will  be  placed. 

3.  After  the  message  is  received  a  new  buffer  address  must  be 
provided  before  the  next  message  arrives.  Otherwise  the  buffer 
may  be  overwritten. 

i».  The  NI  must  be  "turned  on"  before  each  reception.  Otherwise  a 
message  will  be  missed. 

More  detailed  user  information  can  be  found  in  [MORSE80]. 

Abstract  functions  required  by  a  CSMA/CD  network. 

In  the  serial  link  network  described  previously  there  exists  a 
dedicated  link  (or  circuit)  between  every  site  in  the  system.  A  mes¬ 
sage  is  transmitted  to  a  specific  site  by  switching  to  the  link  dedi¬ 
cated  to  that  site.  This  is  a  circuit  switched  network. 

All  sites  in  an  NI  based  network  must  share  one  bus.  Messages  or 
packets  are  placed  on  the  bus.  Each  message  contains  its  destination 
address.  The  destination  site  will  receive  any  packet  which  is  ad¬ 
dressed  to  it.  This  is  a  packet  or  message  switched  network. 

Communications  on  a  NI  (packet  switched)  network  poses  a  dif¬ 
ferent  set  of  problems  than  the  serial  link  (circuit  switched)  case. 
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Since  there  is  no  centralized  oontrol  on  the  NI  bus,  two  or  more 
sites  may  try  to  transmit  simultaneously.  This  will  result  in  colli¬ 
sion  and  destruction  of  packets. 

Regardless  of  the  network  some  procedure  must  be  used  to  guaran¬ 
tee  that  the  a  site  will  be  ready  to  receive  all  messages  transmitted 
to  it.  These  procedures  can  be  categorized  as  flow  control  pro¬ 
cedures.  In  the  serial  link  network  a  connection  had  to  be  esta¬ 
blished  before  a  message  oould  be  transmitted.  Only  one  message  could 
be  transmitted  per  connection  and  only  one  site  oould  establish  a  con¬ 
nection  at  a  time.  Attempts  to  make  a  connection  by  other  sites  are 
reoognized  by  the  destination  site  and  serviced  eventually.  Hence 
there  was  a  flow  oontrol  limit  of  one  message  to  a  site. 

If  a  node  in  the  NI  network  is  servicing  a  message  it  will  not 
recognize  any  other  message  which  might  be  destined  for  this  site. 
There  are  two  basic  solutions  to  preventing  these  "missed"  messages. 
One  method  entails  acknowledging  each  or  a  group  of  messages  frdm  a 
site.  The  other  ensures  that  the  transmitter  will  wait  intil  the  re¬ 
ceiver  is  ready.  Both  techniques  have  distinct  advantages  and  disad¬ 
vantages  which  will  be  discussed  in  the  next  section. 

In  any  event  a  different  set  of  services  must  be  provided  in  the 
communications  subsystem  layers.  The  link  level  must  be  able  to 
detect  collisions  and  provide  this  information  to  higher  levels.  The 
network  level  no  longer  has  to  establish  a  connection  between  sites. 


Messages  are  merely  submitted  to  link  level  procedures  for  transmis¬ 
sion.  Handling  of  oollislons  occurs  at  the  network  level.  The  tran- 
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sport  level  will  be  responsible  for  guaranteeing  message  reception. 

The  techniques  used  to  provide  guaranteed  transport  are  discussed  in 

% 

the  following  section. 

4.5  Implementation  of  abstract  functions  for  a  CSMA/CD 

The  previous  section  described  the  functions  required  by  the 
CSMA/CD  network.  At  the  link  level  two  functions  are  required:  colli¬ 
sion  detection  and  raw  data  transport.  Collision  detection  is  provid¬ 
ed  by  the  network  interfaces.  Collision  handling  protocols  are  pro¬ 
vided  at  the  network  level.  The  current  implementation  of  collision 
handling  entails  resubmitting  the  message  to  the  link  level  if  a  col¬ 
lision  is  detected.  Reliable  message  delivery  is  provided  at  the  net¬ 
work  level.  As  will  be  shown  the  design  of  software  to  provide  reli¬ 
able  delivery  of  messages  is  influenced  by  the  degree  of  reliability 
required  by  the  system. 

Reliability  in  this  case  is  limited  by  the  message  error  rate  of 
the  communications  subsystem.  Since  the  model  of  the  communications 
subsystem  assumes  that  the  hardware  is  reliable,  the  error  rate  of  the 
communications  subsystem  will  be  proportional  to  the  error  rate  of  the 
hardware.  Because  message  errors  are  not  detectable  by  the  kernel, 
any  such  errors  will  cause  the  system  to  fail.  If  the  system  fails 
for  the  latter  reason  the  EPL  programs  will  have  to  be  restarted. 

The  previous  discussion  did  not  mention  the  possibility  of  a  re¬ 
ceiver  "missing  "  an  uncorrupted  message.  Several  basic  techniques 
were  considered  to  solve  the  problem  and  are  discussed  subsequently. 
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4.5.1  Positive  Acknowledgement  Retransmission  (PAR)  schemes. 

Figure  4.3  is  a  timing  diagram  of  a  basic  PAR  protoool.  Most  PAR 
protocols  are  a  variation  on  this  theme  [SUNSHINE81 ) . 

message 


sender 


ACK 

- > 

1  1 

1 

/ _ 

NACK 

1 

1 

1 

1 

1 

message 

receiver 


repeat  until  received 


Figure  4.3  -  PAR  protocol. 

A  message  is  transmitted  to  a  receiver.  If  a  packet  is  received 
correctly  a  positive  acknowledgement  (ACK)  is  returned  to  the  sender. 
This  ACK  merely  implies  that  the  transport  layer  of  the  receiving  site 
has  accepted  the  message.  It  does  not  imply  that  the  destination  pro¬ 
cess  has  received  the  message.  If  the  packet  was  received  in  error, 
the  receiving  site  may  optionally  send  a  negative  acknowledgement 
(MACK).  Since  it  is  equally  likely  for  a  NACK,  ACK  or  a  packet  to  be 
lost  or  corrupted,  this  scheme  in  itself  is  not  completely  effective. 
The  sending  protocol  must  also  use  timeouts.  After  each  packet  is 
transmitted  a  timer  is  started  and  if  an  ACK  is  not  received  before  a 
timeout  period  has  elapsed,  the  data  packet  is  retransmitted.  If  a 
NACK  is  received  before  the  timeout  the  message  car.  be  retransmitted 
sooner  than  the  timeout  period.  If  an  ACK  has  not  been  received  after 
several  retransmissions  the  sending  site  can  take  some  special  action, 
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such  as  notifying  the  user  process. 

Another  situation  which  must  be  handled  by  the  PAR  protocol  is  a 
lost  ACK.  If  an  ACK  is  lost,  the  sending  site  will  retransmit  the 
packet.  This  will  appear  as  a  duplicate  message  at  the  receivers 
site.  The  receivers  protocol  must  be  able  to  differentiate  between 
new  messages  and  duplicate  messages.  A  solution  entails  a  unique 
identifier  attached  to  the  message  by  the  sender.  The  receiver  must 
keep  track  of  the  identifiers  of  packets  successfully  received.  The 
receiver  must  also  check  all  new  packet  I.D.s.  If  the  packet  I.D.  is 
new,  the  I.D.  is  stared  and  an  acknowledgment  (ACK)  is  returned.  If 
the  I.D.  is  a  duplicate  an  ACK  is  returned  and  the  packet  is  discard¬ 
ed.  Sequence  numbers  are  commonly  used  as  identifiers.  The  sender 
oould  sequentially  number  each  message.  The  sequence  number  main¬ 
tained  by  the  receiver  for  duplicate  detection  may  be  considered  the 
left  edge  of  a  "window"  of  acceptable  sequence  numbers.  The  size  of 
the  window  is  the  number  of  packets  the  sender  may  transmit  without 
having  the  first  message  acknowledged.  Any  message  retransmitted  in 
the  window  will  be  detectable  as  a  duplicate. 

4.5.2  Buffered  Message  Approach 

A  goal  of  the  XDCS2  communications  subystem  is  to  provide  the 
same  degree  of  reliability  that  existed  with  the  serial  link  version 
(XDCS1).  To  do  this,  a  mechanism  must  be  provided  for  preventing 
missed  messages.  PAR  protocols  present  one  solution  to  the  problem. 


Another  solution  which  takes  advantage  of  the  timing  characteristics 
of  the  HI  interface,  was  eventually  adopted.  A  specification  of  the 
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NI  ensures  that  after  a  message  has  been  transmitted  each  node  in  the 
system  must  defer  transmitting  another  message  for  a  prespecified  time 
period  (T).  The  receiver  has  T  seconds  to  accept  the  message  and 
prepare  Itself  for  the  next  possible  reception. 

In  XDCS1  messages  were  not  buffered  at  the  receiving  site.  The 
network  protocol  would  transmit  the  message  in  two  pieces.  First  a 
message  header  was  transmitted  which  the  reci ever  could  use  to  deter¬ 
mine  the  destination  process.  If  there  was  mare  text,  the  sending 
site  would  extract  the  data  from  the  source  process  and  transmit  it 
directly  to  the  destination  process.  This  scheme  is  not  possible  in 
the  NI  based  network  for  several  reasons. 

First,  a  message  must  be  transmitted  as  a  oomplete  unit.  The  NI 
interface  is  essentially  a  DMA  device  which  transmits  contiguous 
blocks  of  data.  This  implies  that  the  entire  message  must  be  formed 
in  one  contiguous  buffer  before  it  can  be  transmitted.  Since  the  re¬ 
ceiver  cannot  predict  which  site  will  send  the  next  message  it  cannot 
provide  the  address  of  the  destination  process  to  the  NI.  Therefore  a 
prespecified  buffer  must  be  provided.  After  the  message  arrived  the 
buffer  oould  be  analyzed,  the  destination  process  determined  and  the 
message  transferred  to  that  process.  The  problem  here  is  that  the  ln- 
terarrival  time  of  messages  (T)  is  very  small  (approximately  200  mi¬ 
croseconds).  It  is  not  possible  to  transfer  the  recieved  message  to 
the  destination  process  and  prepare  the  NI  interface  far  the  next  mes¬ 
sage  in  200  mlcroseoonds.  The  only  viable  solution,  then,  was  to 
buffer  messages  at  the  receiving  site  and  perform  the  transfer  to  the 
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destination  process  at  a  later 
A  pool  of  buffers  would  be 


buffers:  |  i  <■ 


I 


tine. 

provided  at  each  site  -  Figure  4.4. 
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Figure  4.4  -  Buffer  pool  for  flow  control  procedures 
The  receiving  protocol  will  merely  make  the  address  of  the  next  avail¬ 
able  buffer  known  to  the  interface  hardware.  The  function  of 
transferring  the  data  from  the  buffers  to  the  processes  will  have  to 
be  provided  by  higher  levels  of  software. 

Given  that  a  receiving  site  oould  switch  buffers  in  T  seconds, 
only  one  problem  remained:  limited  buffer  space.  Due  to  memory  con¬ 
straints  only  a  finite  number  of  buffers  oould  be  provided  for  message 
reception.  A  mechanism  was  needed  to  prevent  more  messages  to  be  sent 
then  there  are  buffers  available.  In  otherwords  a  flow. control  algo¬ 
rithm  was  required. 

4. 5. 2.1  Flow  Control  -  an  overview. 

Flow  control  is  a  set  of  procedures  whose  purpose  is  to  limit  the 
flow  of  message  traffic  in  a  network  [GERL81],  [LAM75] . 

The  technique  developed  in  this  section  is  based  loosely  on  the 
ooncept  of  virtual  circuit  flow  oontrol  [GERL81],  [KLRK80].  A  virtual 
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circuit  is  a  logical  connection  between  two  communicating  entities. 
Before  a  message  can  be  transmitted  the  connection  must  be  "opened". 
When  the  entities  no  longer  need  to  communicate  the  connection  must  be 
"closed".  Once  the  connection  has  been  established  the  sender  is  as¬ 
sured  that  the  receiver  is  prepared  to  accept  a  prespecified  number  of 
messages.  After  this  prespecified  number  of  messages  has  been  sent, 
the  sender  must  wait  until  the  receiver  has  indicated  that  it  is  ready 
to  accept  mare  messages. 

To  implement  this  kind  of  flow  control  in  an  XDCS  system  a 
mechanism  must  be  provided  for  establishing  a  circuit  and  providing 
the  sender  with  an  acknowledgement  that  a  buffer  is  available.  A 
design  criteria  which  also  should  be  met  is  that  any  change  to  the 
operating  system  to  Implement  the  flow  oontrol  algorithms  will  work 
with  both  the  serial  link  network  and  the  NI  network.  The  serial  link 
version  can  be  used  as  a  test  of  the  flow  oontrol  layer  which  uses  a 
"working"  communications  subystem.  The  flow  oontrol  will  be  viewed  as 
a  new  layer  in  the  XDCS  family.  Flow  control  layer  will  be  placed 
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above  the  communications  subsystem  and  below  the  kernel  -  Figure  4.5. 
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Figure  4.5  -  XDCS  system  with  flow  oontrol 
Three  family  members  will  now  exist: 

1.  A  serial  link  network  without  flow  control  -  XDCS1. 

2.  A  serial  link  network  with  flow  control  -  XDCS3. 

3.  The  NI  based  network  -  XDCS2. 

Any  message  which  must  be  transmitted  will  now  be  submitted  to 
the  flow  control  layer.  The  flow  control  layer  will  determine  if  a 
buffer  is  available  at  the  receiving  site.  When  a  buffer  becomes 
available  the  message  will  be  submitted  to  the  communications  subsys¬ 
tem  and  subsequently  transmitted.  The  oommuni cations  subsystem  at  the 
receiving  site  will  be  responsible  for  placing  the  message  in  an 
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4. 5. 2. 2  Flow  Control  implementation  -  Virtual  Circuits. 

The  flow  oontrol  layer  makes  use  of  the  fact  that  kernel  level 
protocols  are  half  duplex  operations.  In  otherwords,  the  kernels  in¬ 
volved  in  a  kernel  call  will  serially  transmit  messages  to  each  other 
until  the  protocol  is  oomplete  (Chapter  3).  For  each  protocol  in  pro¬ 
gress  only  one  buffer  is  needed  at  each  site:  a  transmit  buffer  and  a 
receive  buffer.  After  a  message  of  the  protocol  has  been  transmitted 
the  transmit  buffer  can  be  used  to  receive  a  reply  -  figure  4.6. 

site  A  !  transmit  |  — - network - >j  receive  |  site  B 

|  buffer  |  |  buffer  ! 

*  I 

I 

I 

reply 

Figure  4.6  -  Virtual  Circuit  buffers 

A  virtual  circuit,  then,  consists  of  a  pair  of  buffers:  one  at  the 
kernel  initiating  the  protocol  and  one  at  the  destination  kernel.  Be¬ 
fore  a  protocol  can  begin,  the  flow  control  layer  will  check  to  see  if 
a  virtual  circuit  is  available  between  the  opposing  sites.  If  none  is 
available  the  process  which  initiated  the  kernel  call  will  be  placed 
on  a  waiting  list  until  a  virtual  circuit  becomes  available.  After  a 
virtual  circuit  has  been  granted  to  a  process,  the  protocol  can 
proceed.  The  virtual  circuit  belongs  to  the  initiating  process  until 
the  end  of  the  protocol.  The  destination  kernel  will  use  the  same 
circuit  to  send  replies. 

The  maximum  number  of  messages  a  site  can  send  to  another  site  - 
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the  "flow  limit"  -  corresponds  directly  to  the  number  of  virtual  cir¬ 
cuits  available  to  that  site.  The  greater  the  number  of  virtual  cir¬ 
cuits  which  are  available  the  greater  the  message  traffic  in  the  net¬ 
work. 


In  a  network  with  S  sites  the  number  of  buffers  required  at  each 
site  is  equal  to: 

S  x  2  x  V 

where  V  is  the  maximum  number  of  virtual  circuits  between  each  site. 

4. 5. 2. 3  Plow  Control  Data  Structures 

Figure  4.7  illustrates  the  data  structures  used  by  the  flow  con¬ 
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FIGURE  M.7  -  FLOW  CONTROL  DATA  ABSTRACTION 


VQ_Q  is  a  Queue  of  buffer  addresaes  which  are  available  to  initiate  a 
virtual  circuit.  There  are  FLOW_LIMIT  buffers  for  each  aite. 
MAX_SITES  is  the  maximum  number  of  sites  in  the  network.  If  no 
buffers  are  available  then  a  protoool  can  not  be  started. 

The  routine  START_PROTOCOL  will  take  a  buffer  off  the  VC_Q  and  place 
it  on  the  XMT_Q. 

XMT_Q  is  a  queue  of  buffers  to  be  transmitted  by  the  commmuni cations 
subsystem.  Upon  completing  a  transmission  the  communications  subsys¬ 
tem  will  place  the  buffer  on  the  RECV_Q  if  the  message  was  non  local 
or  on  the  LOCALi_IH_Q  if  the  message  was  local. 

RECV_Q  contains  buffers  which  are  available  for  reception  of  a  message 
by  communications  subsystem.  Upon  reception  of  a  message  the  communi¬ 
cations  subsystem  will  place  the  buffer  on  the  IR_Q. 

Ify_Q  is  a  queue  of  messages  received  by  the  communications  subsystem 
but  not  yet  serviced  by  the  kernel. 

LOCAU_IH_Q  is  a  queue  of  local  messages  which  have  not  been  serviced 
by  the  kernel. 

The  routine  POLI<_IH_Q  will  service  the  input  queues  and  call  the  ap¬ 
propriate  kernel  routines.  Depending  on  which  part  of  the  protocol  is 
being  executed  the  buffer  will  be  returned  to  the  VCjQ  or  the  XMT_Q. 

CUD  points  to  the  buffer  which  has  been  taken  off  the  input  queues. 
The  kernel  will  use  the  information  in  this  buffer  to  oontinue  a  pro¬ 


tocol 
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VCL.VAITING  la  a  linked  Hat  of  proceaa  deacrlptors  which  are  waiting 
for  a  virtual  circuit  buffer. 

Shared  Data  Structure?  aol  critical  Regions 

Because  internet  routines  can  access  concurrently  (with  flow 
control  routines)  certain  flow  oontrol  data  structures,  mutual  exclu¬ 
sion  must  be  established  on  operations  which  effect  these  structures. 
Two  techniques  were  considered. 

The  first  method  would  turn  off  processor  interrupts  upon  entering  the 
critical  regions.  The  interrupt  procedures  would  not  be  able  to  ac¬ 
cess  the  data  structures  ixitil  the  interrupts  were  turned  on  again. 

The  second  method  is  to  use  circular  buffers  to  implement  the  shared 
date  structures.  An  IN  pointer  would  point  to  the  next  place  an  ele¬ 
ment  could  be  placed  in  a  queue  and  an  OUT  pointer  would  point  to  the 
next  element  to  be  taken  off  the  queue.  If  the  concurrent  routines 
access  different  pointers  then  mutual  exclusion  is  ensured.  In  addi¬ 
tion,  if  the  capacity  of  the  queue  is  made  one  larger  than  it  will 

ever  be,  the  queue  will  never  be  full.  Thus  when  IN  is  equal  to  OUT 

the  queue  is  guaranteed  to  be  empty.  The  seoond  method  was  chosen  be¬ 
cause  it  was  more  time  efficient. 

1*. 5. 2. 4  Virtual  Circuit  Procedures 


Five  procedures  will  have  access  to  the  flow  control  data  structures: 

1 .  START_PROTOCOL  | 

2.  CONT__PROTOCOL  >  flow  oontrol  layer 

3.  END_PR0T0C0L  i 

H.  XMIT_INTERRRUPT  | 

>  network  layer 

5.  RECV_INTERRRUPT  i 

ST ART_PROTOCOL  must  be  called  by  any  kernel  routine  which  may  Initiate 
a  protocol:  CREATE, RUN, SEND, REC.RECF.  It  will  first  check  to  see  if  a 
virtual  circuit  is  available.  If  not,  the  process  descriptor  will  be 
queued  on  VC_WAITING.  If  a  virtual  circuit  is  available  a  buffer  will 
be  removed  from  VC__Q  and  the  message  composed  in  that  buffer.  The 
buffer  is  then  placed  on  XMT_Q  for  transmission  through  the  network. 
The  network  layer  can  now  be  called. 

CONT_PROTOCOL  is  called  when  the  kernel  level  protocol  must  be  contin¬ 
ued.  The  same  virtual  circuit  will  be  used  to  transmit  the  text  of 
the  message.  The  message  is  composed  in  the  available  buffer  and 
placed  on  the  XMT_Q. 

END_PROTOCOL  is  called  when  the  last  part  of  the  protocol  is  execut¬ 
ing.  It  will  first  check  to  see  if  there  are  processes  waiting  to  use 
a  virtual  circuit.  If  there  are,  the  waiting  process  is  dequeued  from 
VC_WAITING  and  assigned  the  free  buffer.  The  message  is  then  composed 
and  submitted  to  the  network  layer.  If  there  are  no  processes  waiting 
for  the  virtual  circuit  the  buffer  is  returned  to  VC_Q. 

XMT_IHTERROPT  and  RECLINTERRUPT  can  actually  be  considered  part  of  the 
network  layer.  Upon  completion  of  transmission  or  reception  the  I/O 
devices  must  be  prepared  for  the  next  message.  A  received  message 
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Bust  be  placed  on  the  Ity_Q.  Zn  the  case  of  a  transmission  the  XMT_Q 
is  checked  for  pending  transmissions.  If  the  transmission  was  local 
the  buffer  is  placed  on  the  LOCAL^JCICQ  otherwise  it  is  placed  on  the 
RECV_Q.  This  ensures  that  enough  buffers  will  be  available  to  receive 
the  reply.  In  systems  which  do  not  use  interrupt  I/O  (XDCS3)  the  net¬ 
work  layer  will  queue  and  dequeue  messages  in  a  similar  manner. 


it  .5.2.5  Ramifications  of  Virtual  Circuits  on  the 


1 .  The  virtual  circuit  mechanism  assumes  that  each  protocol  ends  at 
the  site  where  it  is  initiated,  i.e.  there  are  an  even  number  of  mes¬ 
sages: 

start_protocol  - - - — - - - - -> 

|  conti  nue_j>rotocol 


cont_protocol 


< 

i 

i 


> 

conti  nue_protocol 


end  of  protocol  < 


In  previous  versions  of  the  kernel  not  all  of  the  protocols  are  imple¬ 
mented  this  way.  SEND  for  example  does  not  receive  a  positive  ack¬ 
nowledgement  after  a  text  message  has  been  transmitted.  Another  pro¬ 
tocol  with  a  similar  problem  is  RUN. 


2.  Previous  versions  of  the  kernel  do  not  use  a  buffer  to  hold  the 
message.  Messages  were  transported  from  process  to  process.  A  vari¬ 
able  called  CMD  is  a  global  variable  which  contains  all  the  informa¬ 
tion  needed  to  transmit  a  message  of  the  protocol  and  respond  to  one. 
In  particular,  it  oontains  the  source  process  i.d.,  destination  pro¬ 
cess  l.d.,  message  type,  and  the  address  of  additional  parameters. 
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1  1 
I 

I 


The  new  version  of  the  kernel  will  use  a  buffer  abstraction  which  is 
similar  to  CMD  except  the  buffer  will  hold  the  text  of  the  message 
(instead  of  the  address  of  the  text)  and  the  first  byte  will  contain 
the  destination  site  (needed  for  the  N.I.  hardware). 


dest.  |  source  |  function  |  pral 
|  process  |  process!  I 


4. 5. 2. 6  Modified  Kernel  protocols  and  Flow  Control 
Depending  on  which  part  of  the  kernel  protocol  which  is  being  executed 
a  different  flow  control  routine  will  be  initiated.  The  following 
timing  diagrams  illustrate  when  the  different  flow  control  routines 
will  be  executed: 


CREATE 


start_jprotocol 


end_protocol 


start_protocol 


end_protocol 


address  of  act 


name  of  child 


cont_j>rotoeol 


run  time  parameters 


positive  acknowledgement 


cont _j>rotocol 


CHEATE/RUH 


REC/RECF 

i 

i 

i 

i 

I  SEND 


cont_protocol 


cont_protocol 


enquiry  1 

S  — _ —  _ 

1 

1 

1 

i 

ready  to  receive 

I 

message 

! 

1 

1 

1 

1 

1 

1 

positive  ack 

start_protocol 


cont_protocol 


end_protocol 


REC  first  with  no  pending  senders 


56 


cont_protocol 


enquiry 


negative  ack. 


SEND 


start_protoeol 


->  end__protoeol 


REC/RECF 
start_protocol 


enquiry 


cont _protocol 


end_protocol 


message 


positive  ack 


end  protocol 


cont_protocol 


cont_protocol 


SEND  first 
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cont_protocol 


cont _protocol 


<- 

i 

i 

i 

i 


<— 


i 

i 


enquiry 


negative  ack 


message 


positive  ack 


end  protocol 


REC/RECF 

I 

t 

|  start_protocol 

i 

i 


>  end_protocol 


> 


!  cont_protocol 

I 

I 


>  end_protocol 


REC  first  and  pending  senders 
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RECF 


start__protocol(  1)  ! 


end_protocol( 1 )  < — i — 

I 

I 

I 

I 

cont_protocol(2)  I 


cont_protocol(2)  ! 


enquiry( 1) 
enquiry(2) 
negative  ack(1) 


SEND 

start_protocol( 2) 

cont_protocol ( 1 ) 


negative  ack(2) 


message ( 2) 


positive  ack{2) 


|  cont_protoeol( 2) 


->  end_protocol(2) 


SEND  and  RECF  concurrently  with  pending  senders 
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4.5.3  Selection  of  Message  Delivery  Technique 

Virtual  Circuit  flow  oontrol  was  eventually  adopted  as  the  method 
to  provide  guaranteed  message  delivery.  The  reasons  for  this  decision 
were  twofold: 

1.  Virtual  circuit  flow  control  provides  the  same  degree  of  reli¬ 
ability  that  was  provided  in  the  serial  link  version  (XDCS1). 

2.  Since  both  methods  (PAR  and  Virtual  Circuits)  would  work  the 
choice  would  be  based  on  which  method  could  be  implemented  more 
quickly.  The  sooner  the  system  was  implemented  the  sooner  exper¬ 
imentation  could  begin  on  it.  The  judgement  was  therefore  made 
to  implement  the  Virtual  Circuit  technique. 

4.5.M  Summary  -  Comparison  of  Family  members. 

With  the  addition  of  a  flow  control  layer  to  the  XDCS  system  two 
new  family  members  have  been  created.  The  XDCS  family  members  share 
software  and  hardware  in  the  following  way: 

XDCS  family 

i 

• 

GPL 

i 

i 

kernel 

I 


|  flow  control 

!  I 


serial  CSMA/CD  serial 

links  channel  links 

i  i  i 

XDCS1  XDCS2  XDCS3 

Each  leaf  in  the  tree  represents  a  different  family  member.  As  can  be 
seen  in  the  above  figure,  all  family  members  use  the  same  districted 


language  and  the  same  operating  system  kernel.  XDCS1  was  the  first 
member  of  the  family  to  be  created  and  does  not  use  explicit  flow  con¬ 
trol  procedures.  Flow  control  is  implicit  in  the  link  level  connec¬ 
tion  protocol.  Contention  resolution  is  also  handled  in  the  same  con¬ 
tention  protocol.  Since  the  serial  link  network  allowed  the  XDCS1 
communications  subsystem  to  examine  messages  on  a  byte  by  byte  basis, 
messages  oould  be  routed  directly  to  the  destination  process.  Hence 
buffering  of  messages  was  not  required. 

Because  the  CSMA  network  interfaces  were  block  DMA  devices, 
buffers  had  to  be  provided  to  receive  messages.  When  a  message  ar¬ 
rives  it  is  EMA'ed  into  the  buffer.  Since  messages  oould  not  be  exam¬ 
ined  on  a  byte  by  byte  basis  they  could  not  be  routed  directly  to  the 
destination  process.  These  latter  hardware  characteristics  required 
the  implementation  of  the  flow  control  procedures  which  are  documented 
in  the  previous  sections. 

The  XDCS2  and  XDCS3  systems  are  identlcle  except  for  the  network 
hardware.  The  primary  reason  for  implementing  XDCS3  was  to  test  the 
flow  control  concepts  with  a  network  that  was  known  to  be  functioning 
correctly.  The  CSMA/CD  is  a  prototype  design  which  had  not  been 
thoroughly  tested  at  the  time  of  its  acquisition.  Also,  by  using 
XDCS2  and  XDCS3  the  effect  of  two  different  networks  on  the  perfor¬ 
mance  of  the  XDCS  system  oould  be  ascertained  (Chapter  5). 


Chapter  5 


Preliminary  Measurements 
5.1  Purpose  of  Measurements 

Three  systems  are  now  available  to  obtain  measurements  on.  XDCS1 
and  XDCS3  differ  only  in  the  addition  of  flow  control  procedures  in 
XDCS3.  They  both  use  the  same  serial  link  network  and  operating  sys¬ 
tem  kernel.  XDCS2  and  XDCS3  differ  only  with  respect  to  their  under¬ 
lying  networks.  Therefore  the  opportunity  exists  to  answer  the  fol¬ 
lowing  questions: 

1 .  How  much  does  the  overhead  of  the  flow  control  procedures  ef¬ 
fect  the  performance  of  the  XDCS  system? 

2.  The  CSMA/CD  network  uses  a  one  megabit  bus.  The  aerial  links 
provide  a  bandwidth  of  38.2  kilobaud.  Given  systems  with  identi- 
de  kernels  and  communications  subsystems,  does  the  CSMA/CD  based 
system  (XDCS2)  provide  the  same  order  of  magnitude  improvement  in 
Performance  over  the  serial  link  system  (XDCS3)? 
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5.2  Bene  hoar  lea 


The  benchmarks  used  to  answer  the  questions  of  the  previous  sec¬ 
tion  are: 

1.  Commun cations  Subsystem  Bandwidth 

This  benchmark  will  determine  the  bandwidth  seen  by  the  communi¬ 
cations  subsystem.  The  benchmark  consists  of  a  sending  process  at 
site  0  and  a  receiving  process  at  site  1.  One  thousand  messages  are 
transmitted  from  the  sender  to  the  receiver.  The  average  time  to 
transmit  a  40  byte  message  is  determined.  The  experiment  is  executed 
on  the  serial  link  systems  and  the  CSMA/CD  system. 

2.  Process  Bandwidth 

This  benchmark  will  determine  the  bandwidth  as  seen  by  EPL 
processes.  A  sending  process  is  created  at  site  0.  A  receiving  pro¬ 
cess  is  created  at  site  1 .  One  thousand  messages  are  transmitted 
between  the  processes.  The  total  time  to  transmit  1000  messages  is 
measured.  The  total  time  is  divided  by  1000  to  find  the  average  time 
to  transmit  a  message.  The  experiment  is  performed  on  all  three  sys¬ 
tems  (XDCS1 , XPCS2 , XDCS3 )  and  is  repeated  for  messages  of  varying 
length. 

3.  Communications  Overhead  benchmark 

This  program  consists  of  a  pair  of  processes  which  transmit  100 
messages  from  the  sender  to  the  receiver.  By  executing  this  program 
with  1  and  then  2  epu's,  the  network  communications  overhead  can  be 


determined.  If  both  processes  of  the  benchmark  are  executed  by  a 
single  cpu,  the  comm uncations  subsytem  uses  local  memcry  to  transmit 
messages.  If  each  process  is  located  on  a  different  cpu,  the  network 
must  be  used.  When  two  sites  are  used,  the  completion  time  of  the 
benchmark  will  be  increased  by  the  delays  imposed  by  the  network. 
However,  because  the  computations  of  the  EPL  processes  may  be  executed 
in  parallel,  there  may  be  some  improvement  in  performance. 

The  point  at  which  it  becomes  beneficial  to  use  two  processing 
elements  can  be  determined  by  varying  the  amount  of  computation  (per¬ 
centage  of  time  computing  in  EPL)  between  messages  in  the  EPL  program. 

5.3  Results 

1.  Communications  Subsystem  Bandwidth: 

Serial  Link  systems;  32,000  bits/sec 

CSMA/CD  system:  400,000  bits /sec 


2.  Process  Bandwidth  Measurements  (2  CPUs) 


bandwidth 

(bits/sec) 


0  10  20  30  40 


Size  of  Message  (bytes) 


3.  Network  Overhead  Benchmarks 
XDCS1  -  serial  links,  no  flow  control: 


100 

completion 
time  (sec)  10 

1 


1  CPU 


2  CPUs 


50  7  0  90  1  00 

percent  time  in  EPL 


XDCS2  -  CSMA/CD  network: 

1001 

< 

i 

completion  | 
time  (sec)  10! 


1  CPU 


2  CPUs 


50  7  0  90  1  00 
percent  time  in  EPL 


XDCS3  -  serial  links,  flow  oontrol: 


completion 
time  (sec) 


50  70  90  100 


percent  time  in  EPL 
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5\*^Summary  and  Conclusions 

The  oommuni cations  subsystem  bandwidth  measurement  shows  that  the 
CSMA/CD  network  operates  about  12  times  as  fast  as  the  aerial  link 
network.  The  EPL  process  bandwidth  measurement  shows  that  there  is  at 
best  a  twofold  decrease  in  completion  time  by  using  the  CSMA/CD  chan¬ 
nel.  This  indicates  (in  the  ZDCS  system)  that  the  communcations 
software  has  more  impact  on  performance  than  does  the  bandwidth  of  the 
underlying  network. 


The  network  overhead  benchmark  shows  that  when  the  percentage  of 
time  computing  in  EPL  is  greater  than  70  percent  of  the  total  comple¬ 
tion  time,  the  communications  delays  imposed  by  the  network  are  more 
than  offset  by  using  two  processing  elements.  When  the  time  in  EPL  is 
less  than  70  percent  of  the  total  completion  time,  it  is  better  to 
perform  the  computation  on  one  cpu.  This  crossover  point  is  not 
dramatically  changed  by  the  system  used  to  perform  the  measurement. 


Summary 


The  primary  focus  of  this  project  was  to  design  a  communications 
subsystem  for  the  XDCS  system  which  was  based  on  a  CSMA/CD  network. 
Another  goal  was  to  document  the  XDCS  family.  Measurements  were  taken 
to  determine  the  relative  performance  of  the  various  family  members. 
A  desired  side  effect  of  this  project  was  to  develop  a  flexible  system 
in  which  the  various  family  members  could  be  created  by  interchanging 
kernel  implementations  or  communications  subsystems. 

6.1  The  XDCS  Family 

The  characteristics  of  the  XDCS  family  were  described  in  chapter 
two.  Chapter  three  presented  the  implementation  of  a  system  which 
used  these  characteristics  as  a  guideline  for  its  design  (XDCS1).  As 
a  result  of  using  the  concept  of  "layers"  of  software,  new  family 
members  could  be  created  by  changing  or  adding  layers  to  the  XDCS1 
system. 


In  chapter  four  a  new  family  member  (XDCS2)  was  created  by  chang- 
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ing  the  communications  subsystem  layer  and  adding  a  flow  control 
layer.  The  following  figure  illustrates  the  possible  family  members 
which  can  now  be  created. 


XDCS  family 

i 

EPL 

I 

kernel 
/  \ 

/  \ 

/  \ 

/  flow  oontrol 

/  /  \ 

/  /  \ 

Serial  CSMA/CD  serial 

links  channel  links 

*  ■  ! 

i  i  ! 

(XDCS1 )  (XDCS2)  (XDCS3) 

Each  path  in  this  tree  represents  another  family  member.  Each  level 
represents  the  software  or  hardware  shared  by  the  various  family 
members. 


The  flow  control  layer  provided  the  buffers  needed  for  packet 
switched  networks.  Systems  which  use  the  flow  control  layer  are  also 
flexible  enough  to  work  with  circuit  switched  networks  such  as  the 
serial  link  network  of  XDCS1. 


Another  possibility  in  flow  oontrol  systems,  is  the  development 
of  a  different  kernel  level  protocol.  If  messages  can  be  buffered  at 
the  receivers  site,  a  sender  need  not  be  blocked  while  waiting  for  a 
receiver  to  beoome  ready  to  accept  the  message.  The  sender  can 
transmit  the  message  and  Immediately  oontinue  ruining. 


The  price  paid  for  added  flexibility  is  of  course  the  increased 
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overhead  of  the  flow  control  layer.  Also,  message  lengths  are  limit¬ 
ed  to  a  maximum  size  (64  bytes  -  which  can  be  increased  if  needed). 
This  may  be  restrictive  but  currently  is  not  a  problem. 

6.2  Communications  Subsystem  fcr  the  CSMA/CD  network 

The  NI  network  is  a  packet  switched  network  which  required  mes¬ 
sages  to  be  buffered  at  the  receiving  site.  The  virtual  circuit  tech¬ 
nique  (chapter  four)  provided  a  simple  way  to  supply  the  needed  buffer 
space  and  Implement  flow  oontrol  at  the  same  time. 

The  disadvantage  to  the  virtual  circuit  method  is  that  the  mes¬ 
sage  transfer  error  rate  (between  processes)  is  directly  proportional 
to  the  reliability  of  the  underlying  network  hardware.  Experience 
gained  in  this  experiment  showed  that  prototype  hardware  may  not  per¬ 
form  up  to  its  specifications.  In  situations  where  the  reliability  of 
the  network  cannot  be  ascertained,  a  PAR  protocol  should  be  con¬ 
sidered. 

Another  problem  is  that  the  flow  control  algorithm  is  "keyed" 
into  the  kernel  protocol.  A  change  in  the  kernel  protocol  may  ad¬ 
versely  effect  the  operation  of  the  flow  control  layer. 

The  virtual  channel  scheme  does  work.  It  provides  for  buffering 
of  messages  to  transmitted  and  received.  So  far  it  has  been  proven 
successful  for  both  a  serial  link  and  a  CSMA/CD  network.  It  may 
therefore  be  reasonable  to  assume  that  the  virtual  channel  scheme  will 


work  with  other  networks  as  well 
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6.3  Performance 

The  preliminary  measurements  provided  in  chapter  five  indicate 
that  the  CSMA/CD  based  system  performs  as  well  or  better  than  the 
serial  link  versions.  Since  increased  throughput  was  not  a  goal  of 
this  project  the  performance  of  the  CSMA/CD  system  is  at  least  satis¬ 
factory. 

The  level  of  performance  obtained  by  the  CSMA/CD  system  was  not 
an  obvious  result  when  the  added  overhead  of  flow  control  is  con¬ 
sidered.  The  fact  that  the  bandwidth  of  the  CSMA/CD  network  is  12 
times  that  of  the  serial  link  network  and  that  the  performance  of  the 
systems  which  use  these  networks  is  nearly  equal,  indicates  that  the 
protocols  in  the  XDCS  system  have  a  greater  impact  on  performance  than 
does  the  bandwidth  of  the  network. 

6.4  Further  Work 

The  virtual  circuit  flow  control  technique  provided  a  simple 
solution  to  message  buffering  in  a  packet  switched  network.  It  pro¬ 
vides  the  degree  of  reliability  specified  by  the  model  of  a  communica¬ 
tions  subsystem  for  the  XDCS  system.  If  a  more  robust  system  is  need¬ 
ed,  other  techniques  such  as  PAR  schemes  should  be  considered. 

As  stated  in  section  5.1  a  kernel  protocol  can  now  be  considered 
which  is  predicated  on  the  fact  that  buffering  is  available.  A  sim¬ 
plified  kernel  protocol  may  compensate  for  the  added  overhead  of  the 
flow  control  layer. 

Further  extensions  to  the  XDCS  family  can  still  be  implemented. 


For  example  a  version  of  EPL  with  exception  handling  facilities  ex¬ 
ists  [EEB80] .  The  kernel  should  be  modified  to  implement  these  excep¬ 
tion  handling  primitives.  This  may  eventually  lead  to  the  development 
of  communications  subsystem  software  which  can  detect  failures  in  the 
network  and  report  the  failures  to  the  kernel.  The  kernel  can  in  turn 
inform  the  EPL  processes  through  the  exception  handling  facilities. 
Users  of  the  system  will  then  be  able  to  investigate  fault  tolerant 


algorithms. 


Appendix  1  -  Details  of  the  EPL/KERNEL  interface 

This  appendix  describes  the  current  LSI-11  implementation  of  the 
EPL/KERNEL  interface. 

For  all  processes  EPL  expects  the  first  word  of  the  data  segment 
to  be  initialized  to  the  name  of  the  process.  This  must  be  done  by 
the  kernel  when  the  data  segment  is  allocated.  The  EPL  keyword  SELF 
refers  to  this  location.  When  a  kernel  call  is  matte,  the  base  of  the 
data  segment  is  defined  by  the  value  in  rO.  The  value  in  rl  is  a  dis¬ 
placement  into  the  data  segment  where  the  parameters,  needed  by  the 
kernel  call,  can  be  found.  EPL  assumes  that  the  kernel  will  return 
any  results  starting  at  the  latter  displacement.  When  the  operating 
system  kernel  returns  control  to  the  EPL  process,  the  process  assumes 
that  the  success  or  failure  of  the  operation  is  indicated  by  the  value 
in  rl.  A  value  of  -1  indicates  success;  a  value  of  0  indicates 
failure.  Currently  the  operating  system  kernel  assumes  all  operations 
are  successful  and  returns  the  value  -1.  The  parameters  which  are 
passed  differ  for  each  kernel  call  and  are  discussed  subsequently. 

1.  CREATE  -  EPL  requires  that  a  process  be  allocated  memory  at  the 
time  of  its  creation.  This  allocation  takes  the  form  of  two  data 
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structures:  a  process  descriptor  and  a  data  segment.  The  size  of  the 
process  descriptor  is  constant.  The  size  of  the  data  sement  is  depen¬ 
dent  on  the  process  being  created.  (For  mere  details  of  these  data 
structures  see  the  section  -  Kernel  Data  Structures  -  Chapter  2.)  Fig¬ 
ure  A. 1  illustrates  the  data  segments  before  and  after  the  kernel  has 
been  requested  to  create  a  new  process. 
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|  number  of  | 
!  parameters! 
|  expected  | 


* - !  act 


I  act 
i  code 


!< - + 


i  self 


child 
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an  act 


creating  process 


created 

process 


Figure  A.1  Creating  a  process 

Parameters  passed  by  EPL  are  "epu"  and  "act",  "epu"  specifies  the  in¬ 
dex  of  the  epu  assigned  to  execute  the  new  process.  A  negative  value 
indicates  that  the  EPL  program  has  not  specified  an  index.  "Child"  is 
the  name  of  the  newly  created  process.  The  value  of  "child  "  is 
determined  by  the  kernel  and  returned  to  EPL.  "Act"  specifies  the  ad¬ 
dress  of  the  first  executable  statement  of  code  defining  the  newly 
created  process.  It  should  be  noted  here  that  "act"  is  an  address  of 
the  start  of  a  program.  Therefore,  identicle  oopies  of  this  code  must 
exist  at  all  sites  and  start  at  the  same  location  in  memory.  See  ap¬ 
pendix  3  for  memory  layout. 


2.  RUN  -  Figure  A. 2  illustrates  the  process  data  segnents  before  and 
after  the  operating  system  kernel  has  been  called  to  transmit  initial¬ 


ization  parameter  values  to  a  newly  created  process. 
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Figure  A. 2 

The  parameter  "dst"  specifies  the  name  of  the  new  process  waiting  to 
receive  initial  parameters.  The  parameter  "n"  specifies  the  number  of 
parmeters  to  be  transmitted.  The  parameters  p[1]  ...  p[n]  are  the 

values  to  be  transmitted.  The  parameters  p[1]  ...  p[m]  are  the  values 
that  are  received  to  initialize  the  new  process.  The  value  of  "m"  is 
that  specified  in  the  code  segment  of  the  process  (see  figure  A.1), 

3.  SEND  -  Figure  A. 3  illustrates  the  process  data  segment  of  a  sending 


*  ‘ 


process  before  and  after  a  message  has  been  sent  by  the  kernel. 
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Figure  A. 3  -  Data  segnent  sending  process 
The  parameter  "dst"  specifies  the  destination  of  the  message  to  be 
transmitted.  The  parameter  "n"  specifies  the  number  of  par-maters  to 
be  transmitted  to  the  destination  process.  The  parameters  p[1]  ... 

p[n]  are  the  values  to  be  transmitted  to  the  destination  process. 


4.  REC/RECF  -  Figure  A. 4  illustrates  the  process  data  se©nent  before 


76 


and  after  a  message  has  been  received. 


m 

src 


self 


rl 

I 

<-rO 

before  receiving 


p[m] 


PCI] 

src 


self 


<-r0 
after  receiving 


Figure  A. 3  -  Data  segnent  sending  process 
The  parameter  "src"  specifies  the  source  of  the  message  in  the  case  of 
the  RECF  function.  Its  value  is  not  interpreted  by  the  REC  function 
of  the  operating  system  kernel.  When  the  kernel  functions  of  REC  and 
RECF  complete,  the  value  "src"  contains  the  name  of  the  process  that 
sent  the  message.  The  parameter  "m"  specifies  the  number  of  parame¬ 
ters  that  the  process  expects  to  receive.  The  parameters  p[1]  ... 
p[m]  are  the  values  received  by  the  process. 


Appendix  2  -  State  Transitions  For  XDCS1 
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State  Transition  Definitions 


KERNEL  FUNCTION  DESCRIPTION 

1.  createl  -  a  new  process  has  been  created. 

2.  runl  -  the  new  process  has  been  given  it's  runtime  parameters 

and  is  ready  to  begin  execution, 

3.  create  -  a  process  is  creating  a  new  process. 

4.  create2  -  the  creating  process  has  received  the  name  of  the 

created  process  and  is  ready  to  execute  again. 

5.  finish  -  a  process  has  been  terminated. 

6.1  rec  -  a  process  is  prepared  to  receive  a  message  from  any 

sender  and  there  are  pending  senders.  An  enquiry 
will  be  made  to  a  sender's  site. 

6.2  rec  -  a  process  is  prepared  to  receive  a  message  from  any 

sender  and  there  are  no  pending  senders.  No  enquiry 
will  be  made. 

7.1  reef  -  a  process  is  prepared  to  receive  a  message  from  a  specific 

sender  and  there  are  pending  senders.  An  enquiry 
will  be  made  to  the  sender's  site. 

7.2  reef  -  a  process  is  prepared  to  receive  a  message  from  a  specific 

sender  and  there  are  no  pending  senders.  No  enquiry 
will  be  made. 

8.  recv2  -  a  negative  acknowledgement  has  been  received  from  a 

potential  sender. 

9.  run  -  a  creating  process  is  transmitting  the  runtime  parameters 

to  the  created  process. 

10.  done  -  a  process  has  completed  the  transmission  of  data  and  is 

ready  to  begin  execution  again. 

11.  send  -  a  process  is  ready  to  send  a  message.  It  will  issue  an 

enquiry  to  the  receiving  process. 

12.1  send2  -  the  potential  receiver  process  is  not  prepared  to  accept 
a  message. 


A. 
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12.2  aend2  -  the  potential  receiver  process  has  concurrently  made 
an  enquiry  to  the  sender  process.  The  sender  process 
will  transmit  the  message. 

13.  send3  -  a  process  is  now  prepared  to  receive  from  any  process 

and  has  issued  an  enquiry  to  a  site  with  pending  senders. 

Upon  receipt  of  the  enquiry  at  the  sender's  site  the  kernel 
will  choose  a  waiting  sender  and  transmit  its  message. 

14.1  send4  -  a  process  is  now  prepared  to  receive  from  a  specific 

process  and  has  issued  an  enquiry.  Upon  receipt 
of  the  enquiry  the  sending  process  will  transmit 
it's  message. 

14.2  send4  -  a  process  is  now  prepared  to  receive  from  a  specific  process 

and  has  issued  an  enquiry.  The  sending  process  has 
concurrently  issued  an  enquiry  to  the  receiving  process. 

The  sending  process  is  therefore  not  prepared  to  transmit 
it's  message  and  sends  a  negative  acknowledgement  to  the 
receiver  process. 

15.  sendl  -  in  response  to  an  enquiry  from  a  sending  process  the 
receiving  process  has  sent  a  positive  acknowledement. 

The  sender  transmits  it's  message. 

16.1  recvl  -  a  receiver  process  has  accepted  a  message  from  the  sender 

process  and  is  now  prepared  to  execute  again. 

16.2  recv3  -  a  sender  has  made  an  enquiry  and  the  receiver  has  acknowleded 

that  enquiry 


Appendix  3  -  Process  Descriptor  for  XDCS1 


result 

1 

1 

- 

result  of  kernel  call 

pc 

1 

1 

- 

program  counter 

arg 

\ 

1 

- 

address  of  arguments  on  the  stack 

state 

1 

1 

- 

state  of  the  actor 

prev_sndr 

1 

1 

- 

site  of  last  process  to  send  to  this  one 

total_pending 

1 

1 

- 

total  #  of  messages  pending  for  this  actor 

pendi ng[ N0_SITES ] 

• 

• 

• 

I 

1 

1 

1 

- 

#  of  messages  pending  from  site  n 

pred 

1 

- 

link  to  predecessor  in  a  list 

succ 

1 

- 

link  to  a  successor  in  a  list 

name 

1 

1 

- 

name  of  this  actor  (also  base  of  stack) 

stack 

1 

1 

Process  Descriptor  for  XDCS1 


•r aimSfc'Jti**!*— -  ' 


Appendix  4  -  Format  of  messages  for  XDCS1 

This  appendix  describes  the  format  of  the  messages  which  are  used  in 
the  kernel  level  protocol  (Chapter  2).  A  brief  description  of  the  in¬ 
tent  of  each  message  is  given  which  is  followed  by  the  actual  format. 
The  number  of  bytes  is  given  in  parenthesis. 

CREATE  request:  Receipt  of  a  create  request  message  causes  the  kernel 

to  create  a  new  actor  locally.  The  name  of  the  parent  is  passed  so 

that  the  kernel  knows  where  to  return  the  child’s  name. 

function  name  (2) 
name  of  act  (2) 
name  of  parent  (2) 

CHILD  NAME:  The  child  name  message  is  used  to  return  the  name  of  the 

child  to  the  parent  actor. 

function  name  (2) 
name  of  child  (2) 
name  of  parent  (2) 

RUN  request:  A  run  request  message  contains  the  initialization  argu¬ 
ments  for  a  newly  created  actor.  Once  the  child  is  initialized  it  can 
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be  started  ruining. 


function  name  (2) 
name  of  child  (2) 
name  of  parent  (2) 
number  of  arguments  (2) 
arguments  (variable) 

ENQUIRIES:  The  sending  actor's  kernel  passes  an  enquiry  message  to  the 
receiver's  kernel  to  indicate  that  a  sender  is  prepared  to  send  a  mes¬ 
sage.  A  receiver  may  make  an  enquiry  to  see  if  a  sender  is  ready  to 
transmit  a  message. 

function  name  (2) 

name  of  sender  or  receiver  (2) 

name  of  receiver  or  sender  (2) 

ACKNOWLEDGEMENTS:  The  acknowledgement  message  is  the  response  to  an 
enquiry  message.  It  can  be  either  a  positive  or  negative  acknowledge¬ 
ment. 


function  name  (2) 
name  of  sender  (2) 

name  of  receiver  (2) 

*  >  '■  1 

MESSAGE  reception:  This  message  oontains  the  message  expected  by  a  re¬ 
ceiver  and  the  number  of  arguments  expected. 

function  name  (2) 
name  of  sender  (2) 
name  of  receiver  (2) 
number  of  arguments  (2) 
message  (variable) 


Appendix  5  -  Memory  Layout  and  File  Description  of  XDCS1 
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Interface: 


When  an  EPL  process  executes  a  kernel  call,  a  trap  Is  made  to  the 
interface.  The  interface  saves  the  general  purpose  registers  on  the 
system  stack.  The  program  counter  is  placed  in  the  running  actors 
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process  descriptor.  Upon  leaving  the  interface  this  information  is 
restored.  The  interface  is  responsible  for  determining  which  kernel 
call  was  made  and  executing  the  appropriate  oode.  It  will  also  checks 
to  see  if  remote  sites  are  trying  to  send  a  message.  If  so,  then  the 
appropriate  network  software  is  called.  The  code  for  starting  the 
kernel  begins  at  location  1100.  It  initializes  the  system  stack,  pro¬ 
cessor  status  register,  and  calls  system  initialization  routines. 

Measurements : 

This  software  prints  the  results  of  measurements  that  were  in¬ 
stalled  in  XDCS1.  Currently  these  measurements  Include  the  percentage 
of  time  spent  performing  each  kernel  call,  time  spent  in  the  communi¬ 
cations  subsytem,  the  destination  of  messages,  and  the  number  of  bytes 
transmitted  to  each  site.  The  code  to  execute  the  measurement 
software  begins  at  location  1000. 

Act_base : 

Act_base  marks  the  beginning  of  the  EPL  code. 
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